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ABSTRACT 

The  main  thrust  of  this  paper  will  be  centered  around 
the  issue  of  whether  the  Navy  should  restructure  the  Opera- 
tional Test  and  Evaluation  Force  (OPTEVFOR)  along  the  lines 
of  the  Army's  and  Air  Force's  independent  test  agencies.  The 
paper  begins  v/ith  an  overview  of  the  weapon  system  acquisi- 
tion process.  It  then  proceeds  to  demonstrate  how  test  and 
evaluation  (T&E)  fits  into  the  acquisition  process.  Finallyi 
a  description  and  evaluation  of  the  each  of  the  service's 
independent  test  agencies  are  presented. 

The  researcher  suggests  that  the  perceived  problems  are 
not  caused  by  OPTEVFOR's  present  structural  arrangement,  but 
are  the  result  of  OPTEVFOR's  philosophy.  The  final  conclu- 
sion is  that  OPTEVFOR  should  not  change  its  existing  struc- 
ture, instead  it  needs  to  expand  upon  its  current  emphasis 
on  increased  communication. 
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I.    INTRODUCTION 

Weapon  system  acquisition,  the  development  and  procure- 
ment of  military  systems,  is  a  matter  of  widespread  and 
increasing  concern.  Government  Reports,  Congressional 
hearings,  as  well  as  articles  in  the  press,  have  suggested 
that  the  acquisition  process  is  not  working  as  well  as  it 
should.  A  few  examples  of  these  shortcomings  are:  that 
development  fails  to  produce  a  satisfactory  working  design, 
that  major  systems  incur  large  cost  overruns,  and  that  parts 
are  over  priced. 

Testing  of  new  weapon  systems  is  one  of  the  Department 
of  Defense's  (DOD)  key  controls  in  the  process  of  acquiring 
these  complex  and  expensive  systems.  Adequate  test  and 
evaluation  (T&E)  of  these  weapon  systems  is  of  paramount 
importance  to  ensure  maximum  return  for  the  dollars  spent 
to  procure  the  systems. 

Significant  improvements  have  been  accomplished  in  the 
test  and  evaluation  portion  of  the  acquisition  process.  In 
earlier  years,  test  and  evaluation  was  often  accomplished 
when  or  where  time  allowed  and  usually  by  personnel  who  were 
not  trained  in  the  techniques  of  testing.  After  the  July 
1970  Blue  Ribbon  Defense  Panel  Reports,  test  and  evaluation 
policies  were  significantly  altered.  Separate  independent 
test  agencies  were  established  within  each  service.  These 
agencies,  although  organized  differently,  had  one  thing  in 


common,  each  maintained  its  testing  independent  from  the 
developer,  reporting  directly  to  its  own  service 
headquarters.  These  independent  agencies  perform  v;hat  is 
commonly  called  Operational  Test  and  Evaluation  fOT&E).  They 
check  to  see  if  and  how  well  the  system  will  function  in  an 
operational  environment. 

How  Operational  Test  and  Evaluation  is  being  implemented 
has  been  one  of  the  central  issues  identified  in  numerous 
committees,  commissions  and  reports  on  weaknesses  in  weapon 
system  acquisition.  The  issues  addressed  in  these  studies 
have  been  consistently  repeated.  A  report  done  in  February 
1977  by  the  Defense  Science  Board  identified  one 
significant  shortcoming  in  OT&E.  The  report  stated  that  it 
is  very  important  for  all  participants  in  the  acquisition 
process  to  participate  throughout  the  entire  acquisition 
process.  At  the  time  of  this  report  this  interaction  between 
the  developer  and  service  OT&E  organizations  did  not  exist. 
[Ref.  1:  p. 7] 

Directives  and  instructions  have  been  issued  which 
should  have  corrected  some  of  the  major  OT&E  problems.  These 
instructions  however,  have  not  been  effectively  implemented 
in  the  Mavy,  and  this  has  resulted  in  the  Navy's  OT&E 
practices  remaining  essentially  unchanged  with  little 
interaction  between  the  developer  and  the  independent 
operational  test  agency.  This  course  of  action  has  been  one 


of  the  reasons  for  an  increase  in  the  length  of  the 
acquisition  process,  which  in  turn  has  increased  the 
overall    costs    of    the    process    as    well. 

As  a  result  of  the  Tlavy's  reluctance  for  interaction,  a 
question  has  been  asked  about  the  present  structure  of  the 
Navy's  independent  operational  test  agency.  This  question  is 
should  the  Navy  restructure  its  Operational  Test  and 
Evaluation  Force  (OPTEVFOR)  along  the  lines  of  the  Army  and 
the  Air  Force?  Presently  the  Army  and  Air  Force  independent 
oper'ational  test  agencies  act  as  a  manager,  allowing  the 
user  to  do  the  actual  operational  testing.  The  flavy's 
Operational  Test  and  Evaluation  Force  on  the  other  hand, 
acts  as  a  surrogate  for  the  fleet  (user)  and  does  all  the 
Operational  Test  and  Evaluation.  It  is  felt  by  some,  that 
by  restructuring,  the  user's  inputs  could  be  introduced  into 
the  process  on  an  earlier  and  continuous  basis.  The 
contention  is  that  the  restructuring  would  also  enormously 
reduce  the  size  and  cost  of  OPTEVFOR  and  additionally  would 
provide  a  trained  first  fleet  unit  at  the  end  of  the 
Operational  Evaluation  (OPEVAL).  An  additional  potential 
benefit  espoused  would  be  an  earlier  Initial  Operational 
Capability    (IOC).    [Ref.    2] 

The  main  thrust  of  this  thesis  will  center  around  the 
issue  of  whether  the  Navy  should  restructure  OPTEVFOR  along 
the    lines    of    the    Army    and    Air    Force.    However,    before    this 


issue  can  be  properly  addressed,  a  foundation  has  to  be 
laid.  This  foundation  will  consist  of  an  overview  of  the 
weapon  system  acquisition  process,  how  TdE  fits  into  the 
acquisition  process,  and  finally  a  description  and 
evaluation  of  each  of  the  service's  independent  test 
aRiencies. 


II.  DEFENSE  ACQUISITION  PROCESS 

The  issue  of  whether  to  restructure  the  Navy's  Opera- 
tional Test  and  Evaluation  Force  can  not  be  properly 
addressed  without  a  basic  understanding  of  the  DoD's  weapon 
system  acquisition  process  and  how  the  Service's  T&E  program 
fits  into  the  overall  acquisition  scheme. 

The  present  method  of  managing  the  weapons  systems  ac- 
quisition process  in  the  DoD,  emerged  as  a  result  of  a  study 
by  the  President's  Blue  Ribbon  Defense  Panel  (BRDP)  done  in 
1970.  As  a  result  of  recommendations  of  the  BRDP,  the 
Department  of  Defense  Directive  5000.1  was  promulgated  and 
released  in  1971.  The  next  item  to  be  released  in  this 
evolution  was  the  issuance  of  Office  of  Management  and 
Budget  (0MB)  Circular  A-109,  which  was  released  in  1976.  The 
policies  and  guidelines  in  these  documents  form  the  basis 
for  all  the  subsequent  directives  and  instructions  regulat- 
ing the  acquisition  of  weapon  systems.  These  directives  and 
instructions  enable  the  Secretary  of  Defense  (SECDEF),  who 
is  aided  by  the  Joint  Requirements  and  Management  Board 
(JRMB),  formerly  called  the  Defense  System  Acquisition 
Review  Council  (DSARC),  to  guide  and  control  the  development 
and  production  of  major  weapon  systems  through  a  series  of 
acquisition  phases,  milestone  reviews,  and  decision  points. 
The  process  is  illustrated  in  Figure  1   [Ref.  3:p.  6]. 


10 


MISSION   NEED 

VALIDATION 
RECONCILE 


RESOURCES  PRIORITIES 


PRODUCTION 

DEPLOYMENT 

PHASE 


MILESTONES 


0 

I  I    PHASES 


7\JMSNS,  POM 
/    \   OR/ROC 


CONCEPT 

EXPLORATION 

PHASE 


/ 


FULL  SCALE 

DEVELOPMENT 

PHASE 


DEMONSTRATION 

AND  VALIDATION 

PHASE 


Figure  1   Acquisition  Phases  and  Milestone; 
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It  is  the  Secretary  of  Defense  who  makes  the  decision  as 
to  which  new  systems  will  be  designated  a  major  system.  The 
decision  to  designate  any  system  as  major  is  based  upon: 

a.  Development  risk,  urgency  of  need*  or  other  items  of 
interest  to  the  SECDEF. 

b.  Joint  acquisition  of  a  system  by  the  DOD  and  repre- 
sentatives of  another  nation,  or  by  two  or  more  DOD 
Components . 

c.  The  estimated  requirement  for  the  system's  research, 
development.  T&E,  procurement  (production)  and  opera- 
tion and  support  resources.  A  Justification  for  Major 
System  New  Start  (JMSNS)  is  required  for  all  acquisi- 
tions for  which  the  DOD  Component  estimates  cost  to 
exceed  s^200  million  (FY80  dollars)  in  RDT&E  funds  or 
$1  Billion  (FY80  dollars)  in  procurement  (production) 
funds,  or  both. 

d.  Significant  congressional  interest .[ Ref . U  :  pp.  5-6] 
It  should  be  noted  that  all  weapon  systems  acquisitions 

go  through  a  similar  process  to  the  one  outlined  here  for 
major  systems.  The  only  difference  is  that  decisions  are 
made  at  lower  levels  within  the  Services. 

There  are  four  distinct  phases  and  milestones  in  the 
system  acquisition  process.  The  actual  acquisition  of  a 
new  system  begins  with  either  a  decision  to  establish  new 
capabilities  in  response  to  a  technologically  feasible  op- 
portunity, an  identified  deficiency  in  an  existing 
capability,  a  significant  opportunity  to  reduce  the  DoD  cost 
of  ownership,  or  in  response  to  a  change  in  the  national 
defense  policy  [Ref  4:  pp. 3-4].  This  part  of  the  process  is 
called   Milestone  0. 
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By  preparing  a  Justification  for  Major  System  Mew  Start 
(JMSNS),  the  Services  document  the  need  for  the  new  system 
and  submit  it  into  the  Planning.  Programming,  and  Budget 
System  (PPBS)  process  along  with  the  Service's  Program 
Objective  Memorandum  (POM)  for  the  year  in  which  the  funds 
are  requested.  The  Secretary  of  Defense  may  than  sanction 
the  JMSNS  in  the  Program  Decision  Memorandum  (PDM).  This 
action,  designated  Milestone  0,  authorizes  the  Service  to 
initiate  the  program  when  funds  are  available  and  moves  the 
program  into  the  next  acquisition  phase  which  is  called 
Concept  Exploration  (CE). 

In  the  Concept  Exploration  phase,  several  contracts  are 
awarded  to  industry  to  identify  and  investigate  alternative 
system  design  concepts  that  will  satisfy  the  mission  need. 
At  the  conclusion  of  this  phase,  the  Program  Manager  (PM) 
makes  his/her  recommendation  for  those  concepts  which  have 
the  potential  for  further  development  and  evaluation  and 
should  be  carried  forward  into  the  next  phase  of  the  ac- 
quisition process  which  is  called  the  Demonstration  and 
Validation  (DaV)  phase.  This  recommendation  is  documented  in 
the  System  Concept  Paper  (SCP). 

The  SCP  is  submitted  to  the  Joint  Requirements  and 
Management  Board  (JRMB)  and  then  forwarded,  if  approved,  to 
the  SECDEF.  The  SECDEF  uses  the  SCP  to  make  the  Milestone  I 
decision  .    The  authority  to  proceed  is  provided  in  the 
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Secretary  of  Defense  Decision  Memorandum  (SDDM).  Approval 
signifies  a  validation  of  the  requirement  and  is  authoriza- 
tion to  proceed  with  uhe  D^V  phase  with  the  most  promising 
concepts  [Ref  4:  p.  4]. 

An  additional  document  developed  during  the  CE  phase, 
is  the  Test  and  Evaluation  Master  Plan  (TEMP).  The  TEMP  is  a 
short,  concise  master  plan  for  T&E.  Its  purpose  is  to  iden- 
tify all  required  T&E  resourcesM  facilitate  long-range  plan- 
ning, programming,  and  budgeting  including  that  of  adequate 
numbers  of  test  hardware  items  and  specialized  major  range 
and  test  facilities^  ensure  accomplishment  of  adequate  T&EM 
and  eliminiate  redundant  testing.  The  TEMP  forms  the  basic 
contract  between  the  Development  Agency  (DA)  which  is 
responsible  for  the  Development  Test  and  Evaluation  (DT&E) 
and  the  independent  test  agency,  who  is  responsible  for  the 
Operational  Test  and  Evaluation  (OT&E).  I7hile  the  initial 
version  of  the  TEMP,  which  is  required  at  Milestone  I,  will 
lack  many  specifics,  through  a  continuous  revision  process 
the  TEI'IP  will  develop  the  necessary  detail.  [Ref.  3:p.  3-19] 

A  positive  Milestone  I  decision  allows  the  system  to 
enter  the  Demonstration  and  Validation  phase.  The  purpose  of 
this  phase  is  to  further  develop  and  validate  the  alterna- 
tive concepts  to  determine  which  concept(s)  should  progress 
into  the  Full  Scale  Development  (FSD)  phase  [Ref. 3:  p. 1-15]. 
This  phase  involves  demonstration  of  the  technology  to  be 
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used  in  the  system  or  critical  subsystems  to  verify  perfor- 
mance and  the  potential  suitability  of  the  concept  to  fill 
the  mission  need.  At  this  point,  scientific  and  technologi- 
cal development  is  required  to  bring  the  concept  to  its 
fruition . 

It  is  during  the  D&V  phase  that  both  DTiE  and  OT&E  begin 
with  the  test  results  being  used  to  support  the  Milestone  II 
decision.  Identifying  critical  issues  and  areas  of  risk 
that  have  been  addressed  by  test  and  evaluation,  the  Deci- 
sion Coordinating  Paper  (DCP)/ Integrated  Program  Summary 
(IPS)  is  prepared  by  the  Program  Manager.  The  DCP/IPS 
provides  the  test  objectives  and  measures  of  effectiveness 
related  to  the  satisfaction  of  mission  need  and  resource 
requirements  that  apply  to  the  test  activity.  The  PM  submits 
the  document  through  the  JRMB  to  the  SECDEF  for  the 
Milestone  II  decision.  SECDEF  approval  authorizes  the 
program  to  proceed  into  the  Full  Scale  Development  phase 
(FSD) . 

The  purpose  of  the  FSD  phase  is  to  produce  a  fully 
tested,  documented,  and  production-engineered  design  of  the 
selected  concept  [Ref.  3:  p. 1-151.  This  phase  is  divided 
into  three  subphases:  engineering,  prototype,  and  pilot- 
production/  transition  to  production.  It  is  during  this  part 
of  the  acquisition  cycle  that  two  major  evaluations  take 
place.  The  first  is  the  technical  evaluation  (TECHEVAL),  the 
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purpose  of  which  is  to  identify  technical  deficiencies  and 
determine  whether  the  design  meets  technical  specifications 
and  requirements  [Ref.  5:  p. 2].  This  evaluation  is  a  very 
important  facet  of  development  testing,  for  if  the  system 
passes  the  TECHEVAL  it  goes  on  to  the  next  major  evaluation, 
which  is  called  Operational  Evaluation  (OPEVAL).  Given  the 
data  from  the  TECHEVAL,  OPEVAL  proceeds  to  test  the  system 
for  demonstration  of  operational  effectiveness  and  opera- 
tional suitability,  in  addition  to  verification  of  fixes  for 
problems  discovered  in  the  TECHEVAL  [Ref.  5:  p.  4],  These 
formal  evaluations  are  performed  in  order  to  certify  readi- 
ness for  the  production  phase. 

The  final  milestone  decision  point,  Milestone  III,  is 
to  decide  whether  or  not  the  system  is  to  enter  into  produc- 
tion and  deployment.  The  PM  documents  the  results  of  the 
FSD  phase,  including  testing,  and  plans  for  future  testing 
are  assessed  again.  The  PM  then  sends  his  recommendations  in 
an  updated  DCP/IPS  through  the  JRMB  to  the  SECDEF  for  his 
decision.  Normally,  however,  the  Secretary  of  Defense 
delegates  the  Milestone  III  decision  to  the  Secretary  of  the 
branch  of  the  Service  that  is  responsible  for  the  system. 
For  less  than  major  systems,  this  decision  in  some  cases  is 
delegated  to  the  lowest  level  in  the  organization  at  which 
overall  view  of  the  program  rests.  This  is  the  point  were 
test  results  play  their  most  critical  role  because   they 
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assist  the  decision  maker  in  his  decision  whether  to  approve 
or  deny  production. 

An  affirmative  Milestone  III  decision  allows  the 
system  to  enter  the  Production  and  Deployment  Phase.  One  or 
more  contractors  are  awarded  a  production  contract  for 
either  low- rate  or  full  rate  production.  Even  after  the 
system  is  being  producedi  testing  does  not  stop.  The  syster.i 
is  assessed  in  new  environments,  in  different  platform 
applications,  in  new  tactical  applications,  or  against  new 
threats . 

Besides  the  aforementioned  fixed  milestones,  additional 
program  reviews  can  be  held  at  any  point  in  the  program  when 
major  problems  arise.  Host  of  the  information  at  the  reviews 
comes  from  the  test  and  evaluation  results  and  is  used  to 
assess  the  programs  progress  and  its  likely  success 
[Ref  6: p.  88].  Depending  on  the  findings,  a  program  may 
advance  to  the  next  phase,  be  canceled  or  just  held  at  its 
present  phase. 

In  summation,  the  acquisition  process  that  is  presented 
here  is  a  very  simplistic  overview.  This  process,  though 
conceptually  simple,  in  fact  is  extremely  detailed  in  its 
requirements.  Test  and  evaluation  is  an  essential  player  in 
the  Department  of  Defense's  acquisition  process,  in  that 
the  T&E  results  help  to  determine  the  future  outcome  of  a 
program . 
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III.  WHAT  IS  TEST  AND  EVALUATION  ? 

Test  and  evaluation  is  a  fundamental  and  integral  part 
of  the  acquisition  process.  Although  the  terms  "test"  and 
"evaluation"  are  used  jointly,  they  do  represent  two  dis- 
tinct and  unique  functions.  One  definition  of  TEST  is  "  a 
critical  examination,  observation  or  evaluation;  a  trial  or 
group  of  trials"  [Ref.  7:p.  2].  Evaluation,  on  the  other 
hand  concerns  the  review  and  analysis  of  data  produced 
during  testing  or  use  of  the  item. 

The  Department  of  Defense  is  generally  concerned  with 
three  types  of  test  and  evaluation:  Development  Test  and 
Evaluation  (DT&E),  Operational  Test  and  Evaluation  (OT&E), 
and  Production  Acceptance  Test  and  Evaluation  (PAT&E) 
[Ref. 8:  p. 2].  Figure  2  is  a  summary  overview  of  the  acquisi- 
tion process,  and  it  illustrates  where  these  three  types 
of  test  and  evaluation  fit  into  the  acquisition  process 
[Ref. 3:  p.  1-18]. 

A.   DEVELOPMENT  TEST  AND  EVALUATION  (DT&E) 

DT&E  is  that  test  and  evaluation  conducted  to 
demonstrate  that  the  engineering  design  and  development 
process  is  complete,  the  design  risks  have  been  minimized, 
the  system  will  meet  the  specifications,  and  to  estimate  the 
system's  military  utility  when  introduced  [Ref.  9:  p.  7-8]. 
Development  Test  and  Evaluation   is  planned,  conducted  and 
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Figure    2.      Summary   Overview  of    the   Acquisition   Process 
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monitored,  by  the  Developing  Agency  (DA)  of  the  Military 
Departments  or  other  Defense  Agencies,  with  the  results 
being  reported  to  the  responsible  Military  Service  Chief. 

DT&E  is  required  for  all  acquisition  programs.  It  can 
be  broken  down  into  three  major  phases:  DT-I,  DT-II,  and 
DT-III.  The  specific  objectives  of  each  phase  are  developed 
by  the  DA  and  are  published  in  the  Test  and  Evaluation 
Master  Plan  (TEMP). 

DT-I  is  conducted  during  the  Demonstration  and  Valida- 
tion Phase  to  support  the  Milestone  II  decision.  Its  prin- 
cipal purpose  is  to  demonstrate  that  all  technical  risks 
have  been  identified  and  that  solutions  to  the  risks  are  in 
hand  [Ref.  9:p.  7-8]. 

Conducted  during  the  Full  Scale  Development  Phase.  DT-II 
is  used  to  support  the  production  and  deployment  (Milestone 
III)  decision.  DT-II  demonstrates  that  the  design  meets  its 
specifications  in  performance,  compatibility,  suppor- 
tability.  survivability,  safety,  and  human  factors  [Ref.  9: 
p.  7-9]. 

The  Technical  Evaluation  (TECHEVAL)  is  the  final  phase 
of  DT-II.  It  is  conducted  to  determine  whether  or  not  a 
system  is  functioning  in  a  technically  acceptable  manner, 
whether  it  meets  technical  and  design  performance  specifica- 
tions, and  to  determine  if  it  is  ready  for  the  Operational 
Evaluation  (OPEVAL).  The  Developing  Agency  (DA)  has  the 
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primary  responsibility  for  planning  the  test  program  and  oo- 
taining  their  results.  After  the  TECHEVAL,  the  DA  certifies 
that  the  system  is  ready  for  the  Operational  Evaluation 
(OPEVAL).  However,  OPEVAL  may  not  commence  until  the 
Secretary  and  the  independent  test  agency  of  the  concerned 
branch  of  the  service  accepts  the  DA  '  s  certification  of 
readiness  for  OPEVAL,  [Ref.  9:  p.  7-9] 

The  final  phase  of  DT&E  is  DT-III.  It  is  conducted  after 
the  production  and  deployment  decision  (Milestone  III).  DT-- 
III  is  used  to  verify  the  correction  of  design  deficiencies 
found  during  TECHEVAL  or  OPEVAL,  and  that  any  improvements 
that  were  made  to  the  system  are  effective. 

3.   OPERATIONAL  TEST  AND  EVALUATION  (OT&E) 

Operational  Test  and  Evaluation  is  the  test  and  evalua- 
tion conducted  by  an  independent  agency  to  determine  the 
prospective  system's  military  utility,  operational  effec- 
tiveness (including  compatibility,  interoperability, 
reliability,  and  maintainability),  and  logistic  and  training 
requirements  [Ref.  9:  p.  7-91.  Since  OT&E  is  conducted  under 
as  near  to  realistic  operating  conditions  as  possible, 
needed  modifications  to  the  system  do  become  apparent  during 
this  process.  The  operating  personnel,  as  well  as  required 
support  facilities,  are  typical  of  those  expected  to  operate 
and  maintain  the  system  when  it  is  deployed. 


21 


OT&S  can  be  broken  down  into  two  categories:  Initial 
Operational  Test  and  Evaluation  (lOT&E),  which  is  all  OTdiE 
conducted  prior  to  the  producLion  and  deployment  decision, 
and  Follow-on  Operational  Test  and  Evaluation  (FOT&E),  which 
is  all  0T2cE  which  is  conducted  after  the  production  and 
deployment  decision.  0T;2cE  can  also  be  divided  into  four 
phases  (two  in  lOT&E  and  two  in  FOT&E). 

OT-I  is  the  first  phase  of  Initial  Operational  Test 
and  Evaluation  (lOT^E),  and  is  conducted  during  the 
Demonstration  and  Validation  Phase  to  support  the  full-scale 
development  decision  (Milestone  II).  Its  objectives  are  to 
provide  an  early  assessment  as  to  whether  or  not.  based  on 
the  system's  effectiveness,  the  project/program  should  con- 
tinue on  its   development  [Ref.  9:  p.  7-9], 

The  other  phase  of  lOT&E  is  called  OT-II,  and  is  con- 
ducted during  the  Full-Scale  Development  Phase.  The  OPEVAL 
is  the  culmination  of  OT-II.  The  objectives  of  OT-II  are  the 
demonstration  of  the  achievement  of  program  objectives  for 
operational  effectiveness  and  of  operational  suitability. 

The  first  phase  of  Follow-on  Operational  Test  and 
Evaluation  (FOT&E)  is  called  OT-III,  and  is  conducted  after 
the  production  and  deployment  decision.  This  testing  is  done 
before  production  systems  are  available.  Normally  it  is 
done  using  the  same  preproduc t ion  system  that  was  used  in 
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the  OPEVAL.   The  specific  objectives  of  OT-III  include:  test- 
ing of  fixes  or  corrections  that  were  incorporated,  comple- 
tion of  any  deferred  or  incomplete  Initial  Operational  Test 
and  Evaluation  (lOT&E),  and  evaluation  of   any  new  tactics 
development . 

OT-IV  is  the  second  phase  of  FOTgcE,  and  it  is  conducted 
on  the  production  system.  One  of  the  initial  objectives  of 
OT-IV  is  the  demonstration  of  the  achievement  of  program  ob- 
jectives for  production  system  operational  effectiveness  and 
operational  suitability,  especially  reliability,  main- 
tainability and  logistic  supportabili ty .  Additional  objec- 
tives are  OT&E  of  the  system  in  new  environments,  in  new 
applications,  or  against  new  threats. 

C.   PRODUCTION  ACCEPTANCE  TEST  AND  EVALUATION  (PATiE) 

Production  Acceptance  Test  and  Evaluation  is  conducted 
upon  production  items  to  ensure  that  the  items  procured  meet 
the  specifications  of  the  procurement  contract.  Successful 
completion  constitutes  the  recognition  that  the  item 
fulfills  the  requirement  for  which  it  was  produced.  PAT<S:E  is 
the  responsibility  of  the  Developing  Agency  and  begins  in 
the  transition  to  production  section  of  tne  Full- Scale 
Development  Phase  and  continues  through  the  Production  and 
Deployment  Phase  [Ref.  9:  p.  7-9]. 

The  single  most  important  message  that  comes  out  of  the 
test  and  evaluation  evolution  is  that  any  new  acquisition 
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program  must  be  capable  of  meeting  or  exceeding  the  real 
v;orld  goals  expected  of  it  in  the  actual  or  closely 
simulated  operational  environment  prior  to  any  major 
pr'oduction  commitment.  The  information  obtained  by  T  &  E  is 
used  to  give  a  better  understanding  of  the  system's 
capabilities,  to  identify  shortcomings,  to  develop 
improvements,  and  to  assist  in  designing  tiie  system's 
future  replacements.  Additionally,  an  important  facet  of  tiie 
program  is  to  verify  the  ability  to  support  and  maintain  the 
system  and  to  develop  requirements  for  training  personnel  in 
the    operation    of    the    system. 
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IV.  THE  SERVICES'  INDEPENDENT  TEST  AGENCIES 
FOR  OPERATIONAL  TEST  AND  EVALUATION 

Having  established  a  base  from  which  to  work,  each  in- 
dividual service  will  be  reviewed  to  see  how  they  are 
structured  to  handle  the  requirement  for  independent  test 
and  evaluation.  However,  before  this  is  undertaken,  the 
issue  of  what  prompted  the  drive  for  the  independent  test 
agencies  will  be  presented. 

During  the  period  prior  to  the  President's  Blue  Ribbon 
Defense  Panel  Report  (released  on  1  July  1970),  defense  sys- 
tem planners  and  developers  received  little  attention  from 
Congress  on  the  methods  they  used  to  develop  and  test  the 
new  systems.  At  that  time,  most  of  the  controversy  sur- 
rounded the  actual  need  for  the  weapon  system  and  the  cost 
of  placing  the  weapon  system  into  the  defense  inventory.  The 
concern  of  whether  the  weapon  system  could  in  fact  perform 
to  the  designated  specifications  or  was  actually  ready  for 
production  was  very  small  outside  the  Department  of  Defense. 
Most  of  this  lack  of  concern  can  be  attributed  to  a  general 
feeling  of  satisfaction  within  the  Congress  that  the  govern- 
ment research,  development,  test,  and  evaluation  (RDT&E) 
side  had  been  very  successful  since  the  United  States  was 
winning  the  intercontinental  ballistic  missile  and  space 
race.  It  was  during  this  period  in  time  that  the  U.S.  placed 
a  man  on  the  Moon.  Another  factor  that  kept  the  Congress's 
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attention  was  the  war  in  Vietnam.  VJith  the  change  of  Presi- 
dents and  a  change  in  attitudes  toward  economic  conditions 
and  domestic  issues,  the  Congress  intensified  its  scrutiny 
of  Defense  expenditures,  in  particular  the  area  of  major 
weapon  system  acquisition.  It  was  this  intensification  by 
the  Congress  that  brought  about  the  dramatic  change  in  the 
military  services'  test  and  evaluation  policies. 

The  President's  Blue  Ribbon  Defense  Panel  (BRDP)  was  one 
of  tlie  major  driving  forces  behind  the  one  hundred  and 
eighty  degree  change  in  attitudes.  In  the  BRDP  report  the 
Department  of  Defense's  operational  test  and  evaluation 
practices  were  specifically  addressed  as  a  significant 
problem  area.  The  following  excerpt  points  out  the  concern 
of  the  panel : 

There  has  been  an  increasing  desire,  particularly 
at  the  OSD  level,  to  use  data  from  OT&E  to  assist 
in  the  decision-making  process.  Unquestionably  it 
would  be  extremely  useful  to  replace  or  support 
critical  assumptions  and  educated  guesses  with 
quantitative  data  obtained  from  realistic  and  relevant 
operational  testing. 

Unfortunately,  it  has  been  almost  impossible  to 
obtain  results  which  are  directly  applicable  to  the 
decision  or  useful  for  analysis.  Often,  test  data  do 
not  exist.  I7hen  they  do.  they  frequently  are  derived 
from  tests  which  were  poorly  designed  or  conducted 
under  insufficiently  controlled  conditions  to  permit 
valid  comparisons.  It  is  especially  difficult  to 
obtain  test  data  in  time  to  assist  in  decision-making. 
Significant  changes  are  essential  if  OTiS  is  to  real- 
ize its  potential  for  contributing  to  im.portant  deci- 
sions, particularly  where  the  tests  and  the  decisions 
must  cross  Service  lines.  [Ref.  10:  p. 59] 
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Secretary  Packard,  then  the  Deputy  Secretary  of  Defence, 
was  the  driving  force  within  the  DeparT::.]ent  of  Defense  for 
changing  the  test  and  evaluations  policies.  He  personally 
sent  a  series  of  memoranda  to  the  Services  in  which  he 
requested  that  the  Services  arrange  for  OT^E  to  be  managed 
and  conducted  by  an  independent  agency  separate  from  the 
developing  command  and  which  reported  their  results  directly 
to  the  Chief  of  the  Service. 

The  formal  directive  came  in  the  form  of  two  Department 

of  Defense  Directives  (DODDs).  The  first  v;as  Department  of 

Defense  Directive  5000.1,  which  officially  tied  OTiE  with 

the  Joint  Requirements  and  Management  Board  (JRM3),  then 

called  DSARC,  PJilestone   III  decision  point. 

Test  and  evaluation  shall  commence  as  early  as  pos- 
sible. A  determination  of  operational  suitability, 
including  logistic  support  requirements,  will  be  made 
prior  to  large  scale  production  commitments,  making  use 
of  the  most  realistic  environment  possible  and  the  best 
representation  of  the  future  operational  system  avail- 
able. The  results  of  this  operational  testing  will  be 
evaluated  and  presented  to  the  DSARC  at  the  time  of  the 
production  decision.  [Ref.  11:  para.  HI,  C.I] 

The  second  directive  was  DODD  5000.3,  which  provided  ad- 
ditional and  more  specific  test  and  evaluation  policy 
guidance  than  that  of  DODD  5000.1. 

In  each  DOD  component  there  will  be  one  major 
agency  separate  and  distinct  from  the  developing/ 
procuring  command  and  from  the  using  command  which  will 
be  responsible  for  OT&E  and  which  will: 

a.   Report  the  results  of  its  independent  test 
and  evaluation  directly  to  the  Military 
Service  Chief  or  Defense  Agency  Director. 
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c  . 


Recommend  directly  to  its  Military  Service 
Chief  or  Defense  Agency  Director  the  ac- 
complishment of  adequate  OT&E. 
Insure  that  the  OT>iE  is  effectively  planned 
and  conducted  [Ref.128:  para.  IV,  C.I]. 


The  stage  had  now  been  set  with  the  official  Office  of 
the  Secretary  of  Defense   direction  for  the  formation  of  an 
independent   testing  agency  v;ithin  each  service. 

The  Blue  Ribbon  Defense  Panel  Report  should  be  con- 
sidered the  catalyst*  v;hich  resulted  in  a  major  change  in 
how  the  Services  viewed  their  test  and  evaluations  ac- 
tivities. The  most  notable  change  came  in  the  area  of  OT(?cE. 
An  examination  of  how  each  branch  of  the  service  incor- 
porated this  new  philosophy  follows. 

A.   ARMY 

During  the  period  1962-1971,  the  testing  of  systems  and 
equipment  under  development  and  in-process  of  acquisition  by 
the  Army  was  accomplished  almost  totally  within  the  Army 
Material  Command  (AI'IC).  The  Army  Material  Command  assigned 
the  test  responsibilities  to  the  Commanding  General,  United 
States  Army  Test  and  Evaluation  Command  (TECOM). 

As  the  Army's  principal  material  testing  organization, 
the  Test  and  Evaluation  Command  was  assigned  the  basic  mis- 
sion of  providing  the  decision  makers  with  unbiased  indepen- 
dent appraisals  of  Army  material.  The  Army  chartered  the 
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TECOM  to  reduce  the  time  frame  between  design  and  production 
and  to  eliminate  duplication  of  effort  through  integrated 
testing  and  better  coordination. 

To  accomplish  what  the  charter  mandated,  TECOM  employed 
both  engineering  tests  and  service  tests.  Engineering  tests 
were  conducted  to  determine  if  the  system  really  met  the 
material  need.  Service  tests  examined  the  human  inter' face 
with  the  system.  These  tests  determined  the  equipment's 
suitability  for  the  Army's  use  and  for  release  to  produc- 
tion. Actual  operational  units  were  employed  to  conduct 
tests  in  a  tactical  environment  and  in  general  possessed 
many  of  the  characteristics  of  present  OT&E  practices. 
However,  these  tests  were  often  conducted  after  the  manage- 
ment decision  to  produce  the  system  in  numbers  and  even 
after  the  deployment  for  Army  operational  use. 

Since  the  Test  and  Evaluation  Corfimand  was  affiliated 
with  the  Army's  Material  Command  developers,  a  new  and  inde- 
pendent command  was  necessary  in  order  to  meet  the  require- 
ments of  the  Department  of  Defense  Directive.  As  a  result, 
the  Operational  Test  and  Evaluation  Agency  (OTEA)  was  estab- 
lished on  25  September  1972  at  Fort  Belvoir,  Virginia. 
Through  their  charter  from  the  Secretary  of  the  Army.  OTEA 
v/as  given  the  management  responsibility  for  major  and  non- 
major  systems.  [Ref.  13:  pp.  C-1  -  C-3] 
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The  Commanding  General  of  OTEA  reports  to  the  Chief  of 
Staff  of  the  Army.  The  mission  and  functions  of  OTEA  are  as- 
signed by  Array  Regulation  (AR)  10-4,  "U.S.  Army  Operational 
Test  and  Evaluation  Agency".  The  mission  of  OTEA  "is  to 
manage  all  user  testing,  operational  testing  (OT),  force 
development  testing  and  experimentation  (FDTE),  and  joint 
user  testing  directed  by  the  Office  of  the  Secretary  of 
Defense"  [Ref.  14:  p.  2],  The  specific  tasks  which  are  as- 
signed to  OTEA  are  summarized  as: 


a.  Plan,  direct  and  evaluate  the  operational  testing  of 
all  major  and  selected  non-major  systems. 

b.  Coordinate  the  operational  testing  of  other  non- 
major  systems. 

c.  Manage  major  and  coordinate  non-major  FDTE. 

d.  Coordinate  Army  participation  in  the  planning  and 
execution  of  Joint  OT&E. 

e.  Provide  a  strong  focal  point  organization  to  keep 
the  Developing  Agency  (DA)  and  the  Office  of  the 
Secretary  of  Defense  (OSD)  fully  informed  on  the 
Army's  OT&E  needs  and  accomplishments. 

The  original  manning  level  of  OTEA  was  set  at  120  person- 
nel, which  included:  53  officers,  2  enlisted,  and  65 
civilians.  In  1975,  the  strength  was  increased  to  250  per- 
sonnel with  125  officers,  20  enlisted,  and  105  civilians. 
Today,   its  strength  is  256  personnel  with  122  officers,  8 
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enlisted,  and  126  civilians  [Ref.  15].  Figures  3  shov;s 
OTEA's  relationship  within  the  Army's  Test  and  Evaluation 
organization.  Figure  M  is  the  organizational  chart  for  OTSA. 
VJhen  the  Operational  Test  and  Evaluation  Agency  receives 
a  system  from  AMC  for  T&E,  it  either  appoints  its  ov/n  test 
director  and  team  or  tasks  the  Force  Command  (who  is  respon- 
sible for  the  continental  U.S.  Army  Divisions)  along  with 
the  Material  Development  and  Readiness  Couimand  and/or  the 
Training  Doctrine  Command  to  perform  the  testing.  This  deci- 
sion is  made  based  upon  the  size  and  importance  of  the 
program/system  to  be  tested.  OTEA's  role  when  it  retains  a 
system  for  testing  is  to  furnish  three  to  five  of  its  per- 
sonnel to  fill  key  positions  on  the  team  such  as  the  Deputy 
Director  for  the  systems  OT,  the  Chief  Analyst,  the  Chief 
Data  Collector,  and  the  Chief  Controller.  TRADOC ,  repre- 
senting the  user,  provides  the  appropriate  assistants  in  the 
areas  of  conceptual  expertise.  The  remaining  positions  are 
filled  by  the  testing  command.  If  the  program/system  OTaE  is 
tasked  to  an  organization  other  than  OTEA,  OTEA  mainly 
evaluates  the  data  generated  by  the  testing  command.  OTEA 
is  currently  directing/managing  OT&E  for  62  major  and  250 
lesser  category   non-major  systems  [Ref.  161. 
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B.   AIR  FORCE 

In  their  attempt  to  conform  to  the  DOD  requirement,  the 
Air  Force  initially  tried  to  achieve  independence  in  opera- 
tional testing  by  placing  the  full  OT&E  responsibility  on 
the  user  Command.  Even  though  this  was  a  step  in  the  right 
direction,  the  Air  Force's  effort  fell  short  of  its  mark  and 
was  often  subject  to  criticism.  To  correct  this  shortcoming, 
the  Air  Force,  in  October  of  157^.  established  its  indepen- 
dent test  agency  calling  it  the  Air  Force  Operational  Test 
and  Evaluation  Center  (AFOTEC)  at  Kirkland  Air  Force  Base, 
N  e  w  1 1  exico.  AFOTEC  is  a  separate  operating  agency  that 
reports  directly  to  the  Chief  of  Staff  of  the  Air  Force.  In 
addition  to  its  headquarters  at  Kirkland.  AFOTEC  has  four 
additional  permanently  established  detachments  at  Edwards 
Air  Force  Base.  California.  Eglin  Air  Force  Base.  Florida, 
Nellis  Air  Force  Base,  Nevada,  and  Kapaun  Air  Station,  West 
Germany.  Along  with  these  four  permanent  detachments  there 
are  additional  field  test  teams  at  various  designated  sites. 
[Ref .  17:  p. 1  ] 

The  center's  headquarters  and  field  teams  include  opera- 
tional, technical,  analytical  and  test  specialists.  The 
field  test  teams  conduct  their  test  at  the  selected  sites 
and  from  the  results  prepare  the  formal  reports.  These  field 
teams  are  composed  of  personnel  from  the  Air  Force  Test  and 
Evaluation  Center  along  with  personnel  from  the  operating 
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and  support  commands  viho    V'.'ill  eventually  receive  and  utilize 
the  systems  in  every  day  operation. 

The  Air  Force  Operational  Test  and  Evaluation  Center's 
stated  mission  is  "to  rnana^re  the  Air  Force  operational  test 
and  evaluation  program  anci  assess  the  military  utility  and 
operational  effectiveness  and  suitabiJ.  ity  of  major  and 
designed  non-major  systems"  [Ref.  17:  p.1].  The  specific 
tasks  v;hich  are  assigned  to  AFOTEC  are  summarized  as: 

a.  Design,  direct,  analyze,  evaluate  and  report  on 

OT-tE  of  major  and  designed  non-major   Air  Force  sys 

terns  . 

b .  Designate  the  Deputy  Test  Director  and  provide  (with 
augmentation  from  major  corimands)  the  OT&E  team  for 
combined  DTiE/I0T4E  programs  retained  by  AFOTEC. 

c.  Monitor  all  non-major  system  OT&E  conducted  by 
the  major  commands  (  e.g.  TAC ,  SAC,  MAC)  through 
review  and  approval  of  tests  plans  and  review  of 
comments  on  the  test  reports. 

d.  Develop  policy  recommendations  for  Headquarters, 
United  States  Air  Force  approval  and  subsequent  im- 
plementation by  all  major  commands. 

e.  Act  as  spokesman  to  the  Joint  Requirement  and 
Management  Board  (JRMB)  on  matters  pertaining  to 
OT&E. 

f.  Support  Congressional  requests  for  OTc:E  information 
on  weapon  systems  for  which  procurement  funds  are 
beingrequested.  [Ref.  17:  p.1] 

Since  becoming  operational,  the  manning  of  AFOTEC  has 
increased  from  the  initial  authorization  of  208,  to  the 
present  strength  of  677.  This  manning  level  includes  477  of- 
ficers, 90  enlisted,   and  110  civilians.  AFOTEC  is 
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additionally  augmented  by  2500  personnel  from  the  previously 
mentioned  operating  and  support  commands  [Ref.  13]  The 
organization  of  AFOTEC  is  similar  to  that  of  the  Army's 
GTE  A.  Figure  5  shov;s  the  overall  relationship  of  AFOT&E 
within  the  Air  Force,  and  Figure  6  is  the  organization  chart 
of  AFOTEC. 

Air  Force  Operational  Test  and  Evaluation  Center  is 
currently  managing  OT^^E  for  55  major  and  designed  non-major 
pro.c^rams.  In  addition,  the  center  is  presently  monitoring 
240  operational  test  efforts  being  done  by  other  major  USAF 
commands.  [Ref.  19] 

C.   NAVY 

The  Navy  had  an  easier  time  than  its  sister  Services  in 
coming  up  with  a  test  agency  that  was  independent  of  the 
developer,  since  the  Navy  already  had  one  in  existence.  This 
command  is  call  the  Operational  Test  and  Evaluation  Force 
(OPTEVFOR) . 

OPTEVFOR  can  trace  its  origin  to  the  final  months  of 
World  17  ar  II.  During  the  Okinawa  campaign,  the  surface 
forces  of  the  Mavy  came  under  a  sustained  concentrated 
attack  by  Japanese  kamikaze  aircraft.  These  suicide  attacks 
took  a  great  toll  in  ships  and  personnel,  and  a  means  to 
deal  with  the  threat  was  urgently  required.  In  July  of  1945, 
the  Composite  Task  Force,  U.S.  Atlantic  Fleet  was  formed  and 
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tasked  with  the  developaient  and  evaluation  of  r.iethods  to 
combat  such  attacks.  This  uas  the  origin  of  the  f.'avy's 
independent    test    agency.     [Ref.     13:    pp.D-1    -    D-3] 

In  the  follov;ing  years,  changes  were  made  to  the  na;ae  of 
the  command  and  to  the  assigned  mission  and  tasks.  These 
changes  resulted  in  a  wider  scope  of  responsibilities  and 
expanded  capabilities  in  test  and  evaluation.  Additionally, 
a  similarly  structured  command  was  formed  on  the  west  coast 
for    the    Pacific    Fleet. 

To  achieve  the  desired  independence  which  was  suggested 
in  the  Blue  Ribbon  Defense  Panel  report  and  required  in  the 
DOD  directives,  OPTEVFOR  had  to  make  a  shift  in  its 
emphasis.  This  change  in  mission  orientation  was 
accomplished  in  the  period  between  late  1971  and  middle 
1972.  VJhile  the  reorganization  did  not  involve  an 
appreciable  organizational  restructuring  or  an  increase  in 
manpower  allocation,  it  did  require  a  different  and  more 
expensive  mix  of  personnel.  The  Navy's  "nands-on" 
operational  testing  concept  required  operational  officers 
and  highly  skilled  noncommissioned  officers  that  were 
familiar  with  the  current  missions,  tactics,  policies,  and 
procedures.     [Ref.    13:    pp.    D-1    -    D-3] 

The  Operational  Test  and  Evaluation  Force  is  now  under 
the  direct  command  of  the  Chief  of  Naval  Operations  (CMO) 
for    the    conduct    of    OT&E.     The    organizational    relationship 
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that  OPTEVFOR  has  with  the  DOD/MAVY  can  be  seen  in  Figure  7. 
The  command  headquarters  for  OPTEVFOR  is  located  at  the 
Norfolk  (Virginia)  !Javal  Base.  For  the  Pacific  area  of 
operations,  a  separate  staff  under  the  Deputy  COMOPTEVFOR  is 
located  at  the  Naval  Air  Station.  North  Island.  San  Diego. 
California.  The  function  of  the  Deputy  COMOPTEVFOR  is  to  act 
as  the  representive  of  OPTEVFOR  in  matters  in  the  Pacific 
area  and.  when  directed,  with  west  coast  operational 
commands  (e.g.  COMNAVAIRPAC ,  COMNAVSUFPAC ) . 

The  mission  of  OPTEVFOR  is  assigned  by  the  CNO  and  is 
"to  operationally  test  and  evaluate  specific  weapon  systems. 
ships,  aircraft,  and  equipments,  including  procedures  and 
tactics,  where  required;-  and.  when  directed  by  CNO,  assist 
development  agencies  in  the  accomplishment  of  necessary 
Developmental  Test  and  Evaluation"  [Ref.  20:  p.  1]. 
The  specific  tasks  which  are  also  assigned  to  COMOPTEVFOR  by 
the  CNO  are  summarized  as: 


a.  Function  as  an  independent  test  agency  for  OT&E 
under  the  command  of  the  CNO. 

b.  Serve   as  principal  advisor  to  the  CNO  for  all 
Department  of  the  Navy  OTocE  matters. 

c.  Present  results  of  OTdE  to  the  JRMB   at   Milestone 
III  (production  decision)  review  and  to  other  reviews 
as  directed  by  the  CtJO. 

d.  Conduct  operational  tests  on  weapon  systems 
includng  ships  and  aircraft.  Evaluate  operational 
effectiveness,  suitability,  and  capability,  reporting 
the  results  to  the  CNO. 
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e.  Assist    development    agencies    in    DT&E.    including    fleet 
support,    as    required,    reporting    the    results    of    such 
assists    and    an    assessment    of    the    system    tested. 

f.  Review    and    evaluate    the    T^E    planning    for    new    v;eapons 
systems    to    address    and    resolve    critical    issues    and 
report    the    findings    to    the    CNO    [Ref.    20:    p.     1]. 

The  current  authorized  personnel  strength  of  OPTEVFOR 
is     approximately     1U00.     Its  Norfolk    Headquarters    has    268 

personnel,  including  119  officers,  119  enlisted,  and  30 
civilians  [Ref.  21].  This  represents  only  about  one-fifth  of 
the  authorized  positions  in  OPTEVFOR.  The  other  personnel 
are  spread  across  the  subordinate  command  (DEPCOMOPTEVFOR ) , 
organic  aircraft  squadrons  (  VX- 1  ,  VX-4 , VX-5  )  ,  and  the 
Sunnyvale  detachment.  Figure  3  shows  the  current 
organizational    arrangement    of    OPTEVFOR. 

It  is  interesting  to  note  that  when  OPTEVFOR  does  OT&E 
on  a  ship  or  submarine  it  preforms  the  tests  in  a  very 
similiar  manner  to  that  of  the  Army  or  Air  Force.  OPTEVFOR 
usually  sends  a  detachment  of  their  people  to  the 
ship /submarine  to  direct  the  testing,  but  utilizes  that 
unit's  precommissioning  crew  instead  of  there  own  personnel 
for  conducting  operations.  In  the  case  of  an  aircraft  or 
related  hardware,  OPTEVFOR  uses  the  total  in-house  concept, 
using    only    their    own    personnel    to    test    the    item. 

Operational  Test  and  Evaluation  Force  is  presently 
managing      OT^E    for    92    major    and    non-major    systems    [Ref.    20]. 


42 


u 
m 
.c 
u 

c 
o 

•H 
4-1 

fd 

N 

-H 
C 

n3 
Cn 
i^ 
O 

O 
IJ-i 
> 

w 
o 


00 
0) 

p 

en 

•H 
Cm 


43 


In  this  chapter  the  services'  independent  test  agencies 
have  been  reviewed.  Their  organizational  relationship  and 
methods  of  performing  OT&E  have  been  highlighted.  A  summary 
of  these  relationships  and  methods  are  provided  in  Appendix 

B. 
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V.   THE  ADVANTAGES  AND  DISADVANTAGES  OF  THE 
SERVICES'  INDEPENDENT  TEST  AGENCIES 

In  response  to  DoD  directives,  each  service  has  elected 
a  different  approach  in  their  organizational  structure  for 
performing  OTiE.  Both  the  Army  and  the  Air  Force  chose  to 
participate  much  more  closely  with  the  Developing  Agency 
(DA).  Both  OTEA  and  AFOTEC  are  small  organizations  v.-hich 
plan,  direct,  and  control  the  operational  testing,  yet  they 
leave  the  actual  testing  to  the  user  cor.imand,  normally 
utilizing  the  unit  programmed  to  receive  the  first  opera- 
tional system. 

The  Navy,  on  the  other  hand,  has  elected  to  use  the  to- 
tal in- house  concept  in  the  case  of  aircraft  systems'  test- 
ing and  evaluation.  OPTEVFOR  acts  as  a  surrogate  for  the 
user  by  utilizing  their  ovjn  independent  test  facilities  and 
assets  to  conduct  tests  on  the  systems.  Additionally,  OPTEV- 
FOR believes  that  the  involvement  with  the  DA  will  jeopard- 
ize their  objectivity  with  the  system  [Ref.  22]. 

In  trying  to  answer  the  question  as  to  whether  OPTEVFOR 
needs  be  to  restructured,  it  will  be  best  to  address  the 
advantages  and  disadvantages  of  each  service's  independent 
operational  test  agency. 

OPTEVFOR's  strongest  feature  is  their  insistence  on 
total  independence  in  order  to  maintain  their  objectivity  in 
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the  T(!cE  process.  Currently,  OPTEVFOR  insists  that  in  order 
to  maintain  their  independence  or  objectivity,  interaction 
between  OPTEVFOR  and  the  DA  has  to  be  kept  to  a  minimum. 
Stated  another  way,  the  issue  is  independence  versus 
cooperation,  with  OPTEVFOR  placing  more  weight  on  the  side 
of  independence.  [Ref.  22] 

It  is  this  independence  that  is  considered  their 
strength  because  they  do  not  have  to  concern  themselves  with 
the  developer  trying  to  twist  their  arm  in  an  attempt  to  get 
a  favorable  decision.  Additionally,  by  making  themselves 
the  tester  and  not  using  the  operational  commands,  they  are 
able  to  address  the  system  with  more  objectivity  since  they 
do  not  have  a  vested  interest  in  the  system. 

OPTEVFOR's  insistence  on  independence  is  also  their  big- 
gest disadvantage.  This  reluctance  has  been  the  focal  point 
in  numerous  studies,  such  as  the  the  President's  Blue  Rib- 
bon Panel  Report  (BRDP)  in  1970.  The  BRDP  believed  that  it 
is  important  to  perform  OTicE  on  an  operationally  configured 
production  system.  However,  if  the  OT&E  process  only  com- 
mences at  OPEVAL,  it  misses  most  of  the  opportunity  to  in- 
fluence the  product  during  its  development  on  behalf  of  the 
operational  forces.  [Ref. 10:  p.  88]  The  Defense  Science 
Board  on  17  February  1977,  published  a  study  stating  that 
"interaction  among  development  test  and  evaluation  and 
operational  test  and  evaluation,  in  addition  to  the  close 
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contact  v;ith  the  users,  pays  very  important:  divxdends  in 
terms  of  money,  time  and  operational  suitability"  [Ref.  1: 
p.  7].  Both  these  reports,  alon,],  with  others,  indicate 
that  a  working  relationship  between  the  DA  and  the  indepen- 
dent test  agency  is  critical  to  the  acquisition  process. 

Since  OPTEVFOR  does  not  actively  participate  in  nhe 
system,  this  tends  to  create  the  "gotcha"  syndrome 
[Ref.  25].  V/hen  a  system  comes  from  TECHSVAL  to  Oi°EVAL, 
there  is  a  possibility  that  it  may  not  pass  due  to  a  change 
in  OPTEVFOR  test  criteria.  A  weapon  is  developed  to  meet  a 
stated  operational  requirement  (OR).  This  requirement  can  be 
affected  by  funding  reductions,  schedule  delays  and  contract 
disputes,  in  addition  to  changing  threats  ,new  weapons  and 
new  tactics.  As  a  result,  the  DA  develops  a  system  that 
fully  meets  the  requirements  of  the  original  OR,  but  as  a 
result  of  the  aforementioned  ongoing  changes,  it  falls 
short  of  the  current  OTdE  criteria.  Since  OPTEVFOR  does 
not  interface  with  the  developing  agency,  it  is  extremely 
likely  that  the  DA  might  not  be  aware  of  these  changes.  The 
end  result  is  the  system  is  not  ready  for  OPEVAL  due  to 
test  criteria  changes.  It  is  at  this  point  that  the 
"gotcha"  takes  place. 

Another  study  done  in  the  summer  of  198^  by  the  Maval 
Research  Advisory  Committee  (NRAC)  further  illustrates  what 
the  results  can  be  when  there  is  little  or  no  interface 
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b9tv;een  OPTEVFOR  and  the  DA.  The  conirnittee  reviewed  a 
random  sample  of  142  of  OPTEVFOR's  OPEVALS  that  were 
conducted  between  1975  and  1982.  The  results  of  the  study 
showed  that  20%  failed  OPEVAL  outright  and  ^1  % 
conditionally  passed.  These  systems  were  assessed  for  their 
operational  effectiveness  and  suitability.  Of  those  that 
failed,  the  majority  of  them  failed  the  suitability  aspect 
of  the  test.  [Ref.  25]  The  question  which  arises  here  is. 
how  can  a  system  get  this  far  in  the  process  and  then  fail? 
This  is  like  having  a  student  advance  all  the  way  through 
the  education  system  and  then  give  this  student  a  literacy 
test  and  find  out  that  the  individual  is  unable  to  read  and 
wr  i  te . 

In  the  author's  opinion,  the  number  of  failures  in 
OPEVAL  could  be  attributed  to  OPTEVFOR  insistence  on 
maintaining  their  independence.  With  increased 
communication.  OPTEVFOR  could  alert  the  development  agencies 
to  the  changes  in  the  suitability  requirements  much  earlier 
in  the  acquisition  process.  This  in  turn  would  reduce  the 
number  of  failures.  By  waiting  until  just  before  the  system 
is  ready  to  enter  production  to  communicate,  the  only 
recourse  if  a  discrepancy  is  discovered  is  to  fail  the 
system. 

On  the  other  side  of  the  spectrum  .  a  strong  point  of 
both    the    OTEA's    and    AFOTEC's    arrangement    is    their    increased 
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ability  to  conununicate  with  the  DAs.  Instead  of  representing 
the  user  as  in  the  case  of  OPTKVFOR,  botn  OTEA  and  AFOTEC 
believe  in  getting  the  user  involved  at  a  very  early  stage 
in  the  OT&E  process.  OTEA  uses  a  central  group  comprised  of 
their  personnel  and  the  user  to  test  a  systein.  The  Air 
Force,  on  the  other  hand  obtains  the  personnel  needed  to 
test  the  system  from  the  user  and  support  commands  on  either 
a  Temporary  Additional  Duty  (TAD)  or  Permanent  Change  of 
Station  (PCS)  basis.  These  personnel  are  then  sent  to  AFOTEC 
to  support  the  program. 

OTEA  has  recently  implemented  a  new  plan  called 
Continuous  Conception  Evaluation  (C2E),  which  has  advanced 
the  idea  of  continuous  independent  evaluation  through  the 
program's  acquisition  cycle.  Differing  from  OTEA's  previous 
methodology,  C2E  begins  at  a  much  earlier  point  in  tne 
acquisition  process  and  follows  the  system  past  the 
production  phase  of  the  process.  The  data  generated  from 
models,  simulations,  contractor/government  testing,  and 
operating  trials  is  now  being  collected  and  correlated  for 
future  use.  This  information  will  help  to  provide  trends, 
projections  and  inputs  for  existing  and  future  projects. 
[Ref.  23:  p. 6].  Thru  C2E,  OTEA  is  now  in  essence  tracking  a 
major  system  from  inception  to  retirement  from  inventory. 
Figure  9  shows  the  major  projects  that  are  currently  under 
C2E  [Ref.  23:  p.  101. 
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AFOTEC  is  similar  to  OTEA  in  that  it  establishes  a  close 
liaison  with  the  DA.  The  Air  Force's  new  e.riphasis  on  the 
operational  requirement  of  a  system  has  resulted  in  a  even 
closer  working  relationship  between  AFOTEC  and  the  DAs.  The 
DA  is  now  continually  kept  advised  of  any  chan-^^es  in  the 
system's  operational  requirements  by  AFOTEC.  The  obvious 
benefit  is  that  there  are  no  surprise  OR  changes  when  the 
system  enters  OPEVAL.  Additionally,  the  Air  Force,  in  an 
attempt  to  shorten  the  acquisition  process,  is  trying  to  use 
DT&E  to  help  verify  the  operational  performance  of  a  system. 
A  reduction  in  duplicative  testing  can  be  achieved  since 
some  of  these  tests  will  not  have  to  be  repeated  under  OTaE. 
Presently  being  used  on  the  B-1  bomber,  this  approach  is 
considered  a  signifj.cant  change  and  is  still  in  the  very 
early  stages  of   evolution  by  the  Air  Force.  [Ref.  24:  p.  5] 

This  close  working  relationship  of  OTEA  and  AFOTEC  with 
the  DA  and  user  commands  can  shorten  the  acquisition 
process.  3y  alerting  the  system  to  changes  and,  in  the  case 
of  the  Air  Force,  reducing  duplicative  efforts  in  testing 
the  total  time  required  to  field  a  system  is  reduced. 

The  underlying  reason  behind  the  establishment  of  the 
independent  operational  test  agency  (ITA)  was  to  assure  the 
independence  in  the  testing  of  a  system,  riowever,  one  of 
the  biggest  disadvantages  that  can  result  from  too  close  a 
working  relationship  between  the  ITA  and  the  DA/user  command 
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is  a  loss  of  objectivity.  It  is  very  hard  to  maintain 
objectivity  when  so  much  time  has  been  spent  with  the  people 
developing  the  system.  Botn  OTZA  and  AFOTEC  have  to  contend 
with  pressures  from  not  only  the  DA,  but  also  the  user, 
both  of  whom  are  trying  to  get  the  system  into  production. 
The  DA  wants  the  system  to  enter  production  because  this 
makes  him  look  good.  The  user  wants  the  hardware  since  it  is 
perceived  to  be  better  than  that  which  he  has  presently.  As 
a  result,  he  wants  the  system  to  do  well  in  testing  as  this 
would  expedite  the  fielding  of  the  new  system.  If  the  ITAs 
bends  to  their  wishes,  the  result  could  be  a  system  entering 
production  that  might  require  extensive  after-production 
modifications  to  make  it  function  properly.  Any  savings  in 
time  and  costs  v/hich  were  gained  by  the  ITAs  interfacing  are 
now  removed  by  the  additional  tiaie  and  costs  required  to  fix 
the  system's  shortcomings. 

In  summation,  this  chapter  has  illustrated  that 
OPTEVFOR's  strongest  point,  their  insistence  on 
independence,  is  also  their  weakest  feature.  In  their  effort 
to  insure  their  objectivity,  OPTEVFOR  has  missed  the 
opportunity  to  influence  the  system's  developi:ient  on  behalf 
of  the  operational  forces.  Additionally,  their  lack  of 
cooperation  can  add  to  the  length  of  the  acquisition 
process.  By  not  keeping  the  DAs  appraised  of  changes  in 
the   OR,   it   is  possible  that  a  system  might  have  to  go 
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baclc  through  development  to  cor^rect  the  shortcomings 
introduced  by  the  new  requirements.  On  the  other  nand, 
OTEA's  and  AFOTEC's  best  feature  is  their  strong 
relationship  with  the  DA.  This  relationship  can  correct 
OPTEVFOR's  most  prevalent  shortcomings,  however,  it  in 
itself  can  introduce  problems  if  carried  too  far.  resulting 
in  the  release  of  a  system  that  is  not  yet  ready  for 
production. 
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Vi.   CONCLUSIONS/RECOMMENDATIONS 

Before  we  put  a  weapon  system  in  the  hands  of  our 
troops  we  should  ensure  that  it  performs  its  combat  mis- 
sion, not  that  it  simply  meets  contract  specifications. 
.  .  .  American  service  personnel  have  a  right  to  ex- 
pect that  the  governr.ient  will  provide  them  with  the 
necessary  tools  to  do  their  job.  [Ref.  26:  p.  23] 

This  statement  made  by  Representive  C.  E.  Bennett,  is  an 
excellent  summation  of  V'/hat  is  expected  from  OT&E  in  the  ac- 
quisition process.  The  purpose  of  the  independent  test 
agencies  is  to  determine  whether  the  weapon  system  fulfills 
the  desired  function  in  an  operational  environment. 

In  the  introduction,  a  statement  from  the  Summer  1984 
NRAC  report  indicated  that  restructuring  would  enormously 
reduce  the  size  and  cost  of  OPTEVFOR.  In  addition,  the 
report  stated  that  the  user's  inputs  could  be  introduced 
into  the  process  on  an  earlier  and  continuous  basis  [Ref. 2]. 
The  first  part  of  this  statement  is  appealing,  however  the 
preceding  analysis  of  operational  test  agencies  indicates 
that  there  is  strong  reason  to  believe  that  changes  in 
operating  policy,  not  the  actual  structure  of  OPTEVFOR, 
should  be  made. 

The  present  manning  levels  of  OTEA  and  AFOTSC  are  in  ac- 
tuality about  the  same  or  even  greater  than  OPTEVFOR.  This 
analysis  of  the  manning  levels  is  based  on  counting  the  per- 
sonnel that  OTEA  and  AFOTEC  utilize  from  other  commands  for 
support  of  their  testing.  By  comparing  these  numbers  against 
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OPTEVFOR's  total  complement  of  officerG,  enlisted  and 
civilians,  it  becomes  evident  that  the  Army  and  Air  Force 
OT&E  manning  effort  is  not  less  than  the  iJavy's.  For  ex- 
ample, AFOTEC ' 3  level  of  2500  supporting  personnel  is  al- 
most double  that  of  OPTEVFOR  total  in-house  complement  of 
1300  personnel. 

Considering  the  costs  of  those  services  supplied  oy  the 
other  supporting  commands,  it  appears  that  operating  costs 
of  each  ITA  are  cor.iparaole.  In  essence,  there  is  very 
little  difference  betvv'een  the  size  and  cost  of  each  ITA  if 
the  supporting  commands  for  OTEA  and  AFOTEC  are  included  in 
the  analysis.  The  actual  cost  of  each  ITA  and  its  supporting 
command  was  beyond  the  scope  of  the  research  for  this 
thesis,  however,  the  potential  value  of  this  information 
merits  further  study. 

The  statement  that  the  users  inputs  need  to  be  provided 
earlier  and  on  a  continuous  basis  can  be  viev^ed  as  a  result 
of  OPTEVFOR's  philosophy  and  not  its  existing  structure.  Un- 
til recently,  OPTEVFOR  has  been  resistant  to  the  idea  of 
interacting  more  closely  with  the  DAs,  so  that  it  could 
maintain  objectivity.  The  importance  placed  by  OPTEVFOR  on 
maintaining  objectivity  has  changed  somewhat  witn  the  new 
commanding  officer  (COMOPTEVFOR).  An  examination  of  the 
findings   of  the  Summer  1934  ;jRAC  reoort  reveals  a  change  in 
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relationship  between  OPTEVFOR  and  the  DAs  [Ref.  25].  It  ap- 
pears that  the  new  COMOPTEVFOR  endorses  increased  liaison 
with  the  DAs. 

In  the  past,  when  a  system  arrived  for  the  OPEVAL 
that  did  not  meet  the  test  criteria  it  \s;ould  immediately 
fail.  Now,  if  a  problem  is  discovered,  instead  of  failing 
the  system  outright,  COIIOPTEVFOR  will  call  his  counterpart 
at  the  DAs  and  inform  him  that  the  system  is  not  doing  well. 
This  allows  the  DAs  to  remove  the  system  from  OPEVAL  and 
correct  the  deficiencies.  This  new  approach  not  only  keeps 
the  problems  in-house  and  away  from  the  public,  but  it  also 
fosters  greater  cooperation. 

This  cooperative  relationship  should  not  begin  just  at 
the  time  and  point  that  a  system  enters  OPEVAL.  It  is  not 
enough  to  just  advise  the  DAs  that  the  system  is  not  doing 
well  and  needs  some  correction.  The  L'avy  needs  to  strongly 
encourage  the  earJ.y,  continuous,  and  positive  participation 
of  all  key  players  as  described  in  OPNAVINST  3960.10.  The 
difference  between  the  operational  test  requirements  and 
that  of  development  testing  mandates  closer  coordination 
between  the  Navy's  independent  test  agency  and  its  develop- 
ing agencies.  These  organizations  need  early  and  clear 
definition  of  the  test  objectives,  criteria  and  the  stand- 
ards. Once  these  conditions  can  be  agreed  upon,  they  must 
be  kept  current  throughout  the  program  to  minimize  the 
necessity  for  repeat  development  and  testing.  Failure  to  do 
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30  can  result  in  unnecessary  delays  and  higher  costs  in  the 
acquisition  of  fleet-ready  weapon  systems. 

Another  area  outside  the  scope  of  this  research  is  the 
cost  associated  with  having  to  send  a  systen  back  through  a 
part  of  the  acquisition  process  because  it  fails  OPEVAL. 
There  are  undoubtedly  increased  costs,  however  the  magnitude 
of  this  problem  was  not  pursued.  This  subject  has  potential 
value  in  determining  the  real  efficiency  of  the  OTjcS  effort 
and  merits  further  investigation. 

A  loss  of  independence  in  operational  test  is  not  being 
advocated  nor  is  inference  being  raade  that  objectivity  is 
not  important.  The  real  key  to  effective  and  efficient  OTaE 
is  meaningful  communication  among  the  participants  in  the 
acquisition  process.  If  the  test  requirements  are 
coordinated  and  discussed  by  all  parties,  then  conflict  may 
be  reduced  to  minor  proportions  or  eliminated  altogether. 

OPTEVFOR  should  not  change  its  existing  structural 
arrangement,  since  it  been  has  demonstrated  that  tnis  was 
not  where  the  problem  exists.  Instead.  OPTEVFOR  needs  to 
expound  upon  the  new  movement  of  increased  communication 
v;ith  the  DAs.  The  end  result  of  this  increased 
communication  will  be  a  vastly  improved  independent  test 
agency  that  may  ameliorate  the  potential  of  an  inadequate 
system  reaching  the  fleet.   The  IJavy   can  neither   afford 
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the  development  of  systems  that  do  not  meet  the  fleets 
needs,  nor  can  they  afford  the  publicity  that  results  in  a 
produced  system  that  fails  in  the  fleet. 


APPENDIX  A:   GLOSSARY  OF  ACRONYMS  A,\'D  ABBREVIATIONS 


ACAT 
AFOTEC 

AFR 

AFSC 

AMC 

AR 

ASARC 

ASD(A&L) 

ASN(R&D) 

BIS 

BRDPR 

CDEC 

CNO 

COMOPTEVFOR 

CSA 

CSAF 

CTP 

D&V 

DA 

DCP 


Acquisition  Caoegory 

Air  Force  Operational  Test  and  Evaluation 
Center 

Air  Force  Regulation 

Air  Force  Systems  Command 

Army  Material  Command 

Army  Regulation 

Army  System  Acquisition  Review  Council 

Assistant  Secretary  of  Defense  (  Acquisition 
and  Logistics) 

Assistant  Secretary  of  the  Navy  for  Research 
and  Development 

Board  of  Inspection  and  Survey 

Blue  Ribbon  Defense  Panel  Report 

Combat  Development  Experimentation  Command 

Chief  of  Navy  Operations 

Commander,  Operational  Test  and  Evaluation 
Force 

Chief  of  Staff  Army 

Chief  of  Staff  Air  Force 

Coordinated  Test  Program 

Demonstration  and  Validation 

Developing  Agency 

Decision  Concept  Paper  (NAVY)  or  Development 
Concept  Paper  (ARMY) 
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DCP/ICP        Decision  Coordinating  Paper/  Integrated 
Coordinating  Paper 

DCSRDA         Deputy  Chief  of  Staff  Research  Development  and 

Acquisition 

DDT&E  Deputy  Director  of  Defense  Research  and 

Engineering.  Test  and  Evaluation 

DDR&E  Director  of  Defense  Research  and  Engineering 

DEPCOMOPTEVFOR  Deputy  Commander  Operational  Test  and  Evaluation 

Force 

DoD  Department  of  Defense 

DoDD  Department  Of  Defense  Directive 

DOT&E  Director,  Operational  Test  and  Evaluation 

DSARC  Defense  Systems  Acquisition  Review  Council 

DT  Development  Testing 

DT&E  Development  Test  and  Evaluation 

FDTE  Force  Development  Testing  and  Experimentation 

FOT&E  Follovvf-on  Test  and  Evaluation 

FYTP  Five  Year  Test  Plan 

GAO  Government  Accounting  Office 

ILS  Integrated  Logistics  Support 

IOC  Initial  Operational  Capability 

lOT^cE  Initial  Operational  Test  and  Evaluation 

JMSNS  Justification  of  Major  Systems  Hew  Start 

JRMB  Joint  Requirement  and  Management  Board 

LOI  Letter  of  Instructions 

MAC  Military  Airlift  Command 

MAJCOM  Major  Command 
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MASSTER 

NAVAIR 

OPEVAL 

OPNAV 

OPNAVINST 

OPTEVFOR 

OR 

OSD 

OT 

OT&E 

OTEA 

PATocE 

PCS 

PDA 

PDM 

PM 

PHP 

POM 

PPBS 

RAM 

RDT&E 

RDT3cE 

ROC 

SAC 

Modern  Army  Selection  Systems  Test  Evaluation 
Review 

Naval  Air  Systems  Conmand 

Operational  Evaluation 

Office  of  the  Chief  of  Mavy  Operations 

Office  of  Chief  of  Naval  Operations  Instruction 

Operational  Test  and  Evaluation  Force 

Operational  Requirement 

Office  of  the  Secretary  of  Defense 

Operational  Testing 

Operational  Test  and  Evaluation 

Operational  Test  and  Evaluation  Agency 

Production  Acceptance  Test  and  Evaluation 

Permanent  Change  of  Station 

Program  Decision  Authority 

Program  Decision  Memorandum 

Program  Manager 

Program  Management  Plan 

Program  Objective  Memorandum 

Program,  Planning  and  Budget  System 

Reliability,  Availability,  Maintainability 

Research,  Developr.ient ,  Test  and  Evaluation 

Research,  Development,  Test,  and  Evaluation 

Required  Operational  Capability 

Strategic  Air  Command 
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SAM 

SCP 

SDDM 

SECDEF 

SECMAVINST 

SOR 

SPO 

TAG 

TAD 

T3cE 

TD 

TDP 

TECKEVAL 

TECOM 

TEMP 

TRADOC 

USACDC 

USD  (A) 

VX 


Systems  Acquisition  Management 

System  Concept  Paper 

Secretary  of  Defense  Decision  Memorandum 

Secretary  of  Defense 

Secretary  of  the  Navy  Instruction 

Specific  Operational  Requirement 

System  Prograni  Office 

Tactical  Air  Command 

Temporary   Additional  Duty 

Test  and  Evaluation 

Test  Directive 

Technical  Development  Plan 

Technical  Evaluation 

Test  and  Evaluation  Command 

Test  and  Evaluation  Master  Plan 

Training  and  Doctrine  Command 

U.S.  Army  Combat  Developments  Command 

Under  Secretary  of  Defense  for  Acquisition 

Air  Test  and  Evaluation  Squadron 
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APPENDIX    B 


ARMY/NAVY/    AIR    FORCE    COMPARISON 


US   AIR    FORCE 


US   ARMY 


US    NAVY 


1.     Designation     of 
major    field    agency 


responsible 
OT&E: 


for 


Air  Force  Opera- 
tional Test  and 
Evaluation  Center 
(AFOTEC)  HQ  AF  TEST 
AND  EVALUATION  CEN- 
TER 

Kirkland  Air 

Force  Base 

N e w  M exico  87115 

Autovon:  2^4-0545 

Commercial:  (505) 

264-0545 


U  S  Army  Operational 
Test  and  Evaluation 
Agency  (OTEA)  CG,  HQ 
U  S  ARMY  OPERATIONAL 
TEST  and  EVALUATION 
AGENCY 

Falls  Church  VA. 

Autovon:  289-2228 

Commercial:  (703) 

756-1254 


Operational  Test  and 
Evaluation  Force 
(OPTEVFOR)  CO  M - 
MANDER,  OPERATIONAL 
TEST  and  EVALUATION 
FORCE 

Norfolk  VA.  2351 1 
Autovon:  564-5337 

Commercial :  ( 804 ) 
444_5337 


2.  Manning  Level: 
(authorized) 


Military 
Civilian 


567 
1  10 


3.  Reporting  Chain: 


Military 
Civilian 


130 
126 


Reports  direct  to  Reports  direct  to 
Chief  Staff  of  the  Chief  Staff  of  the 
Air    Force.  Army. 

4.  Principal 
direct Ives /regula- 
tions concerning 
OT&E: 


a.  AFR    80-14     "Test 

and  Evaluation" 

b.  AFR      23-36    "Air 

Force  Operational 
Test  and  Evaluation 
Center    (AFOTEC)" 


a.  AR  70-10  "Test 
and  Evaluation 
During  Development 
and     Acquisition    of 

Material" 


Military   238 
Civilian   30 


Reports  direct  to 
Chief  of  Navy  Opera- 
tions . 


a.  SECNAVINST  5000.1 
"System  Acquisition" 

b.  OPNAVINST  3960.10 
"Test  and  Eval- 
uation" 
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5.  Field  activities 
directly  under  OT&E 
Test   Agency: 

All  facilities  sup- 
port and  personnel 
support  provided  by 
desip,ned    MAJCOMS. 


b.  AR  10-M  "US  Array 
Operational  Test  and 
Evaluation    Agency" 


U  o  n  e  .  OTEA  con- 
tributes deputy  test 
director  and  3  to  5 
key  personnel  to 
test  at  the  command 
conducting    test. 


c   .  0  P  N  A  V  I  N  S  T 

54^0.47  "Mission  and 
Function  of  Opera- 
tional Test  and 
Evaluation    Force" 


a .  Three  Air.  Test 
and  Evaluation  Sqdns 
(VX-1 ,U,and    5) 

b.  Deputy  COMOPTEV- 
FOR    San   Diego 

c.  New  London  Test 
and  Evaluation 
Detachment 


6.  Comparison  of 
characteristics  of 
OT&E: 


a.  Development  of 
optimum  tactics, 
techniques,  proce- 
dures   and    concepts. 

b.  Evaluation  of 
reliability,  main- 
tainability, and 
operational  effec- 
tiveness  and 
suitability . 

c.  Testing  under 
realistic  opera- 
tional    conditions. 

d.  Accomplish  lOT&E 
prior  to  first  major 
production    decision. 

e.FOT&E  subsequent 
to  receipt  of 
production    items. 

f.  lOT&E  will  not 
obviate  the  need  for 
FOT&E. 


a.  Representative 
user    troops 

b.  Realistic  opera- 
tional   environment 

c .  Provide  data  to 
estimate  military 
utility,  operational 
effectiveness,  and 
operational 
suitability  includ- 
ing compatibility 
interoperability  . 
reliability, 
availability  main- 
tainability in- 
tegrated logistics 
support  and  training 
requirements . 

d.  lOT&E  examines 
the  hardware  or  com- 
ponents to  provide 
an  indication  of 
utility  and  worth  to 
the    user. 


a.  OT&E  par- 
ticipated  in  or  per- 
formed by  opera- 
tional   personnel. 

b.  Focuses  on 
operational  effec- 
tiveness  and 
suitability  includ- 
ing reliability 
compatibility,  in- 
teroperability, 
maintainability  and! 
sup portability. 

G.  Development  o f J 
optimum,  operational] 
tactics . 

d.       lOT&E       ac- 
complished    prior    to 
JRMB     III     or     com- 
parable    CIJO     majori 
production    decision] 
point . 
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g.F0T3cE  focuses  on 
the  operational  and 
employment  aspects 
of    the    system. 

AFR    80-14 


e.  Examination  of 
equipment  prior  to 
initial  production 
decision . 

f.  FOTaE  final  check 
prior  to  acceptance. 
AR    70-10 


e.  Assessment  of 
operational  effec- 
tiveness and 
suitability. 

f.  FOT&E  conducted 
in  operational  en- 
vironment by  opera- 
tional personnel 
using  production 
systems . 

g .  Refine  tactical 
employment  doctrine 
and  requirements  for 
personnel  and  train- 
ing. 

op:jAVir;sT  3960.10 


7.  Authority  for  as- 
signment of  OT&E 
projects : 

a.  Chief  Staff  of 
the    Air    Force    (CSAF) 

b.  AFOTEC  may 
recommend  to  CSAF 
those  OT&E  projects 
which  it  or  MAJCOMS 
"should    conduct. 


8.  Master  test  docu- 
ment for  specific 
(major  or  selected 
non-major)  lOT&E 
program: 

Section  5  (Test  and 
Evaluation)  of 
Program  Management 
Plan    (PMP). 


Deputy  Chief  of 
Staff  Research 
Development  and  Ac- 
quisition determines 
the  major  and  non- 
major    systems. 


Coordinated 
Program    (CTP) 


Test 


a.  Chief  of  Naval 
Operations    (CNO) 

b  .  CMC  may 
authorize  COMOPTEV- 
FOR  to  further  as- 
sign projects  to 
other  commands  wnen 
advantages  are  ap- 
parent. 


Test    and    Evaluation 
Master    Plan    (TEMP) 


9.    Views   on   combined 
DT&E/IOT&E: 


a. Operating  and 
supporting  commands 
participation  is 
mandatory  during 
development    testing. 


a.  The  OT&E  process 
will  be  independent 
and  should  normally 
be  separate  from  the 
DT&E    orocess. 


a  .  OPTEVFOR  pa  r- 
ticipates  in  the 
TECHEVAL      (DTocE) 


which     is 
conducted 


planned  . 
by    the 
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b.  Operational  in- 
puts will  be  util- 
ized in  ( test )  plan- 
ning documents 
developed  by  the  im- 
plementing   Command. 

c.  To  maximum  pos- 
sible degree.  lOTuE 
will    be    accomplished 

completed  by  operat- 
ing personnel  of  the 
Operational/  Sup- 
porting Commands 
using  realistic 
operational  environ- 
ment . 

d.  Where  adequate 
test  data  can  be 
secured  from  com- 
bined DT&E/IOT&E, 
combined  tests  may 
be  employed.  Added 
considerations  for 
combined  testing  are 
costs  and  time  (AFR 
30-14) . 

e.  Separate  evalua- 
tions required  from 
Operating/Supporting 
Commands  in  Produc- 
tion   Decisions . 


However,  it  may  be 
combined  where 
sepat^ation  causes 
delay  involving  un- 
acceptable risk  or 
unacceptable  ac- 
quisition  costs  (AR 
70-10) . 

b.  Testing  is 
usually  phased  as  DT 
I.  II,  and  III  and 
OT  I,  II,  and  III. 
Usually  OT  I  is  run 
in  combination  with 
DT  I.  Army  attempts 
to  keep  OT  II  and 
III  and  DT  II  and 
III    separated. 

c.  Each  phase  of  OT 
testing  results  in  a 
separate  evaluation 
by  OTEA,  timed  to 
provide  evaluation 
to  to  the  decision 
body  (JRMB)  at  deci- 
sion  milestones. 

d .  DT  II ,  OT  II  . 
and  DT  III  are  con- 
ducted  on  all 
development  items/ 
systems  as  a  mini- 
mum. 


Developing  Agency 
with  OPTEVFOR  in- 
puts. OPTEVFOR 
provides  independent 
operational  assess- 
ment   to    the    CiJO. 

b.  OPTEVFOR  is 
responsible  for 
planning,  conduct- 
ing, reporting 
OPEVAL ,  conduc  ted 
after  TECHEVAL  and 
prior  to  Production 
Decision    milestone. 


10.  Document  chain 
in  acquisition 
system: 


a  .  RO 
states 
0  p  e  r 
Caoabil 

b'.  Ac 
tives    ( 

c  .  Pro 
Documen 

d.  DCP 
provide 

e  .  P  M 
Test    Di 


C     (formally 
a     Required 
a    t    i   o   n  a   1 
ity) 

tion     Direc- 
early    PMD) 
gram   Advocacy 
ts 

(OT^cE    inputs 
d    by    AFOTEC) 
D     -     Initial 
rective    (OT<iE 


a.  ROC 

b.  Charter  for  spe- 
cial   task    force 

c.  OTEA  evaluation 
plan 

d.  Program  Advocacy 
Document 

e.  DC?  (OTiE  inputs 
provided    by    OTEA) 

f .  Final  Report  of 
Special  Task  Force 
(contains    plan    for 


a.  SOR  (formally 
states  a  Specific 
Operational 
Requirement) 

b.  Technical 
Development  Plan 
(TPD)-  (program  ad- 
vocacy document  in- 
cluding OT&E  inputs 
by    OPTEVFOR) 

c.  OPTEVFOR  evalua- 
tion plan 
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inputs  provided  by 
AFOTEC) 

f .  Test  objectives 
annex    to    the    PMD 

g.  HO.     USAF     Test 
Directive    (Drafted 
by         AFOTEC) 
h     .  AFOTEC 
Commander's    Estimate 

i.  AFOTEC  Test  Plan 

j.  Program  Manage- 
ment Plan  (contains 
DT&E,  OT&E  tests 
plans ) 

k.  Interim  Test 
Reports  (interim 
evaluation ) 

1.  Final  Test 
Report 

m     .  AFOTEC 

Commander's  Summary 
(Final  independent 
ops    evaluation) 


OTo:E  provided  by 
OTEA) 

g.  OTEA  or  TRADOC 
Outline   Test    Plans 

h.    Test    Plans 

i.  Development  Plan 
(contains  Coor- 
dinated Test  Program 
of  all  DT  and  OT 
testing  of  the 
system) 

j.  OT  I/DT  1  Test 
Reports  (independent 
operational  evalua- 
tion   by    OTEA) 

k.  OT  II/DT  II  Test 
Reports  (independent 
operational  evalua- 
tion   by    OTEA) 

1 .  OT  III/DT  III 
Test  Reports 
(independent  opera- 
tional evaluation  by 
OTEA) 


d.  DCP  (OT^^E  inputs 
provided  by 
OPTEVFOR) 

e.  Test  and  Evalua- 
tion Master  Plan 
(TEMP)-  (contains 
DTScE  and  OT^cE  plans 
for  a  specific 
program 

f.  Project  ii  aster 
Plan  (  P  M  P  )  - 
(contains    the    TEMP) 

g.  TECHEVAL  Plan 
(the  developing 
agency's  plan  for 
DT&E) 

h.  TECHEVAL  Report 

i.  OPTEVFOR's  inde- 
oendent  assessment 
of    TECHEVAL 

j.  OPEVAL  (OT&E) 
PLAN 

k.    OPEVAL    Report- 
(independent    opera- 
tional   evaluation    by 
OPTEVFOR) 


11.     Commands    con- 
ducting  OT: 


a.  AFOTEC-  (all 
major  and  selected 
nonmajor    systerus) 

b.  Designed  AF 
MAJCOMS-  (nonmajor 
systems ) 


a.  OTEA-  (all  major 
and  selected  non- 
major    systems) 

b.  Combat  Develop- 
ment Experimentation 
Command  (CDEC)- 
(nonmajor    systems) 

c.  Modern  Army 
Selected  System 
Evaluation  and 
Review  (MASSTER)- 
(nonmajor    system) 

d.  Other  Major  Com- 
mands (nonmajor 
systems ) 


COMOPTEVFOR 
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1  2 .  Command  s  per- 
forming evaluation 
of  OT&E  directed  by 
Service   HQ: 


The  command  conduct- 
ing OT  performs  the 
evaluation  subject 
to  review,  approval, 
and  further  evalua- 
tion   by    AFOTEC. 


OTEA     performs     the 
evaluation . 


COMOPTEVFOR  performs 
the  evaluation. 
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